Search for: All records

Contrastive learning has served as a powerful framework in the early development of vision–language models (VLMs), demonstrating remarkable effectiveness in learning generalizable representations and establishing itself as the foundation for many state-of-the-art systems. However, despite these advances, its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world settings. Such imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is still lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve differently across three stages of training, with distinct dynamics for majority features, minority features, and the noise. We further show that minority features diminish neurons’ representational capacity, increase the need for more complex architectures, and impair the separation of ground-truth features from noise. These findings offer new theoretical insights into how data imbalance shapes learning in contrastive frameworks and serve as an early step towards principled modifications for developing more robust and unbiased representations. 

Contrastive learning has served as a powerful framework in the early development of vision–language models (VLMs), demonstrating remarkable effectiveness in learning generalizable representations and establishing itself as the foundation for many state-of-the-art systems. However, despite these advances, its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world settings. Such imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is still lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve differently across three stages of training, with distinct dynamics for majority features, minority features, and the noise. We further show that minority features diminish neurons’ representational capacity, increase the need for more complex architectures, and impair the separation of ground-truth features from noise. These findings offer new theoretical insights into how data imbalance shapes learning in contrastive frameworks and serve as an early step towards principled modifications for developing more robust and unbiased representations. 

While Transformers rely on a distinctive attention mechanism, the recent emergence of Mamba and other selective state space models (SSMs) offers a strong alternative. These models incorporate attention-like mechanisms with hardware-aware efficiency and a unique selection strategy, yet their theoretical properties remain poorly understood. In this work, we present a first-step theoretical analysis of the selection mechanism in Mamba. We study a simplified single-layer Mamba block trained with gradient descent on structured data containing both label-relevant and irrelevant tokens. Our results show that the gating vector dynamically aligns with label-relevant features while negating irrelevant ones, formalizing its role as an implicit feature selector. Moreover, we prove that training achieves guaranteed generalization, with explicit bounds on sample size and convergence rate. These f indings offer principled insight into when and why Mamba’s selection mechanism enables efficient learning, offering a theoretical counterpoint to Transformer-centric explanations of generalization.